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FOREWORD 


This  project  was  conducted  in  response  to  a  request  from  the  Naval  Air  Systems 
Command  (NASC)  (NAVAIR-413E)  under  contract  number  N66001-79-C-0295  with  Instruc¬ 
tional  Science  and  Development,  Inc.  NASC  is  developing  a  handbook  for  cost  effective 
selection  of  instructional  media  using  a  four-step  approach:  (1)  gather  a  representative 
set  of  instructional  objectives  from  the  P-3  aircrew  training  curriculum,  (2)  choose  a 
media-selection  model,  (3)  exercise  the  model  with  respect  to  training  effectiveness  and 
cost,  and  (4)  incorporate  the  model  into  the  handbook. 

This  report  describes  the  representative  set  of  objectives  from  the  P-3  curriculum, 
three  candidate  media  selection  models,  and  the  results  of  applying  the  model  chosen  by 
NASC  to  the  representative  objectives  with  respect  to  the  criteria  of  training  effective¬ 
ness.  In  accordance  with  NASC  guidance,  cost  criteria  were  not  addressed  in  applying  the 
model. 

The  contracting  officer's  technical  representatives  were  Joseph  C.  McLachlan  and 
Walter  F.  Thode. 


JAMES  F.  KELLY,  JR. 
Commanding  Officer 


JAMES  J.  REGAN 
Technical  Director 


SUMMARY 


Problem 


Media  selection  can  be  a  difficult  task  during  the  development  of  military  training 
programs.  One  facet  of  the  media  selection  process  involves  the  evaluation  of  the 
effectiveness  of  different  instructional  media.  In  the  absence  of  objective  data  obtained 
from  controlled  experimental  studies,  a  variety  of  techniques  are  used,  all  of  which  are 
based  on  subjective  interpretation.  As  the  requirement  to  select  media  increases,  it  is 
necessary  that  current  techniques  be  evaluated  to  determine  which  provide  the  greatest 
levels  of  objectivity. 

Objective 

The  objectives  of  this  study  were  to  survey  state-of-the-art  media  selection 
methodologies,  to  evaluate  the  techniques  currently  being  applied  to  naval  air  training 
programs,  and  to  present  three  of  the  techniques  to  the  Naval  Air  Systems  Command 
(NASC),  who  would  select  one  for  implementation  and  incorporation  into  a  media  costs 
handbook. 

Approach 

Media  selection  methodologies  were  surveyed,  and  the  following  three  were  selected 
to  represent  the  state-of-the-art:  (1)  the  Training  Effectiveness  and  Cost  Effectiveness 
Prediction  (TECEP)  technique,  (2)  a  computerized  technique  using  logic  similar  to  TECEP, 
and  (3)  small-group  process  called  the  "DELPHI"  method.  The  latter  was  included  by 
direction  from  NASC  because  of  specific  interest  in  exploring  its  possibilities  as  a 
technique  that  is  as  efficient  and  as  valid  as  the  other  techniques,  and  because  of  its 
perceived  potential  for  minimizing  subtle  biases.  After  a  review  of  the  three 
methodologies,  NASC  selected  the  "DELPHI"  method  for  test  case  evaluation. 

Results 


The  team  of  experts  evaluating  the  "DELPHI"  method  lacked  sufficiently  structured 
procedures  to  be  successful  and  efficient  in  selecting  media.  They  required  a  better 
definition  of  ground  rules  and  assumptions  than  was  provided  by  the  "DELPHI"  method  to 
solve  the  problems  inherent  in  the  media  selection  process.  Subsequent  attempts  at 
structuring  centered  on  the  definition  of  the  media  hardware,  the  media  attributes  that 
were  useful  in  training,  the  meaning  of  the  objectives,  the  use  of  the  media,  and  the 
scales  for  ranking  the  media.  During  the  course  of  the  9-hour  session,  a  number  of 
discussions  were  held  that  revealed  a  deep  divergence  of  views  among  members  of  the 
group,  all  of  whom  are  respected,  experienced  professionals  in  the  field.  The  team 
evaluated  only  seven  objectives  for  media  selection  during  the  session  and  never  reached 
total  agreement  even  on  those  seven. 

Conclusions 


The  results  indicate  that  any  future  use  of  the  small-group  process  as  a  media 
selection  technique  will  require  more  specific  rules  than  those  used  during  the  test  case 
evaluation.  Further,  the  more  rules  and  assumptions  stated,  the  closer  the  "DELPHI" 
method  resembles  the  more  algorithmic  techniques  surveyed.  The  conclusion  reached  was 
that  the  "DELPHI"  method,  in  fact,  suffers  from  the  same  deficiencies  as  any  other  media 
selection  technique,  even  when  the  ground  rules  and  assumptions  have  been  specifically 
stated. 
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Recom  mendatio  ns 


Until  critical  decision  data  are  available  from  additional  media  selection  research, 
media  selection  techniques  should  include  the  rules  and  assumptions  to  be  employed  if  the 
results  are  to  be  interpreted  meaningfully. 
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INTRODUCTION 


Problem 


Media  selection  can  be  a  difficult  task  during  the  development  of  military  training 
programs.  One  facet  of  the  media  selection  process  involves  the  evaluation  of  the 
effectiveness  of  different  instructional  media.  In  the  absence  of  objective  data  obtained 
from  controlled  experimental  studies,  a  variety  of  techniques  are  used,  all  of  which  are 
based  on  subjective  interpretation.  As  the  requirement  to  select  media  increases,  it  is 
necessary  that  current  techniques  be  evaluated  to  determine  which  provide  the  greatest 
levels  of  objectivity. 

Background 

In  the  early  1970s,  the  Training  Analysis  and  Evaluation  Group  (TAEG)  investigated 
the  adequacy  of  existing  techniques  for  choosing  instructional  media  and  embarked  on  a 
program  to  eliminate  deficiencies.  In  a  review  of  ten  existing  techniques  (Braby,  1973), 
TAEG  identified  the  following  deficiencies  that  make  the  selection  of  instructional  media 
for  military  training  programs  a  difficult  task: 

1.  The  procedures  are  inexact. 

2.  The  selection  criteria  are  too  general  for  identifying  specific  media  for  specific 
training. 


3.  Some  procedures  are  too  general  to  address  the  instructional  requirements 
sufficiently. 

4.  Most  procedures  are  too  complicated. 

5.  Most  procedures  cannot  be  adapted  easily  to  the  range  of  military  training  needs. 

TAEG  identified  several  reasons  for  the  deficiencies.  First,  the  nature  of  military 
jobs  provides  an  intrinsic  difficulty  in  selecting  an  optimal  media  mix;  that  is,  job 
performance  frequently  involves  a  combination  of  procedural,  perceptual,  motor,  and 
knowledge  components,  perhaps  including  decision  making,  that  must  be  integrated.  Thus, 
the  learning  task  may  not  consist  of  discrete  components  easily  associated  with  media 
hardware  attributes.  Likewise,  it  is  often  difficult  to  associate  one  specific  medium  with 
the  learning  task,  given  a  variety  of  attributes  of  concern.  Often,  the  use  of  a  medium 
and  the  resulting  quality  of  the  courseware  determines  a  successful  media  application  just 
as  much  as  the  unique  attributes  of  the  medium  itself. 


Instructional  hardware  has  evolved  continually,  and  a  wide  range  of  equipment  is 
available  for  consideration.  However,  unbiased  information  on  the  instructional  hardware 
is  seldom  available.  There  is  little  evidence  for  empirically  determining  which  media 
hardware  attributes  are  important  to  specific  learning  tasks,  and  little  evidence  that  is 
related  to  how  well  specific  media  selection  techniques  work. 

The  initial  survey  of  media  selection  techniques  led  TAEG  to  an  ambitious  program 
for  developing  a  more  appropriate  technique  for  choosing  instructional  media.  The  result 
was  the  Training  Effectiveness  and  Cost  Effectiveness  Prediction  (TECEP)  technique 
(Braby,  Henry,  Parrish,  5c  Swope,  1975),  which  consists  of  a  three-step  procedure:  (1) 
classify  training  objectives  into  groups  and  define  the  appropriate  learning  strategies  for 
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each  group  of  objectives,  (2)  identify  the  media  capable  of  supporting  the  strategies,  and 
(3)  estimate  the  costs  of  alternative  media.  The  technique  was  field  evaluated  in  several 
cases  and  research  has  continued  at  TAEG  on  some  aspects  of  it.  Also,  the  TECEP 
technique  has  been  incorporated  into  Phase  III  of  the  Interservice  Procedures  for 
Instructional  Systems  Development  (NAVEDTRA  106A,  1975). 

Despite  the  efforts  expended  on  TECEP  and  other  such  programs,  Naval  Air  Systems 
Command  (NASC)  personnel  judged  that  many  deficiencies  still  exist  in  the  selection  of 
media  appropriate  to  naval  air  training  programs. 

Objective 

The  objectives  of  this  study  were  to  survey  state-of-the-art  media  selection 
methodologies,  to  evaluate  the  techniques  currently  being  applied  to  naval  air  training 
programs,  and  to  present  three  of  the  methodologies  to  NASC,  who  would  select  one  for 
implementation  and  incorporation  into  a  media  costs  handbook.  NASC  was  particularly 
interested  in  determining  whether  a  small-group  process  called  the  "DELPHI”  method  was 
effective. 


APPROACH 

NASC  assigned  five  tasks  to  accomplish  the  media  effectiveness  evaluation: 

1.  Review  the  candidate  media  to  be  used  in  the  media  costs  handbook. 

2.  Develop  a  representative  sample  of  training  objectives  to  be  approved  by  NASC. 

3.  Conduct  a  literature  search  on  media  effectiveness  measures. 

4.  Identify  and  describe  three  media  effectiveness  methodologies. 

5.  Conduct  a  test  evaluation  of  one  media  effectiveness  methodology  specified  by 
NASC. 

The  first  four  tasks  supported  the  fifth  task— to  conduct  a  test  case  evaluation.  Task 
1  determined  the  list  of  candidate  media  to  be  used  during  Task  5.  Task  2  developed  a 
representative  sample  of  objectives  sufficiently  large  to  exercise  the  media  effectiveness 
technique  to  be  evaluated  in  Task  5.  Tasks  3  and  4  produced  a  description  of  three 
alternative  methodologies  for  selecting  training  effective  media.  NASC  reviewed  this 
description  prior  to  specifying  the  media  effectiveness  technique  to  be  evaluated  in  Task 
5.  The  tasks  are  described  in  the  following  paragraphs. 

Candidate  Media 


Since  the  purpose  of  the  media  costs  handbook  was  to  provide  cost  indices  for  media 
that  satisfy  training  requirements  for  cognitive  rather  than  psychomotor  skills,  no  trainers 
or  simulators  were  considered  as  candidate  media.  However,  three  general-purpose 
computer-assisted  instruction  (CAI)  systems  were  included.  The  following  list  of  media 
was  specified  for  use: 

1.  Lecture  (with  limited  visual  support). 

2.  Programmed  text. 
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3.  Linear  text  (e.g.,  Naval  Air  Training  and  Operating  Procedures  Standardization 
(NATOPS)  Manual). 

4.  Workbook. 

5.  Programmed  filmstrip  (e.g.,  Beseler  Cue/See). 

6.  Slide  with  sound. 

7.  Random-access  slide. 

8.  Videotape  cassette  (e.g.,  Sony  U-Matic). 

9.  Videodisc. 

10.  Time-shared  interactive  computer-controlled  instructional  television  (TICCIT) 
CA1. 

11.  Programmed  Logic  for  Automatic  Teaching  Operation  (PLATO). 

12.  General  Electric  Training  System  (GETS). 

13.  Microfiche  (random-access  and  random-access  with  microprocessor  control). 
Training  Objectives 

The  sample  of  training  objectives  was  selected  from  the  P-3  Fleet  Readiness  Squadron 
aircrew  training  program.  The  P-3  aircrew  training  program  consists  of  approximately 
6,000  cognitive  learning  objectives  and  a  larger  number  of  "hands-on"  objectives.  It 
covers  three  versions  of  the  P-3  aircraft  (B  MOD,  C,  and  C  Update),  found  in  Fleet 
Readiness  Squadrons  on  both  coasts  (VP-30  and  VP-31).  The  learning  objectives  are 
organized  into  courses  for  seven  aircrew  positions:  pilot,  flight  engineer  (FE),  naval  flight 
officer  (NFO),  sensor  station  operators  1  and  2  (SS  1/2),  sensor  station  operator  3  (SS  3), 
ordnanceman,  and  communicator. 

Given  the  size  of  the  training  program  and  scope  of  the  study,  the  number  of 
objectives  for  analysis  was  limited  by: 

1.  Excluding  learning  objectives  presented  in  device  sessions  using  trainers, 
simulators,  or  aircraft,  since  the  media  costs  handbook  was  designed  for  media  that 
satisfy  cognitive,  rather  than  "hands  on,"  training  requirements.  This  focused  attention  on 
the  6,000  cognitive  learning  objectives. 

2.  Eliminating  the  aircrew  positions  of  ordnanceman  and  communicator,  since  these 
courses  did  not  provide  many  objectives  suitable  for  the  candidate  media.  This  decreased 
the  pool  to  3,550  learning  center  objectives  for  five  remaining  aircrew  positions. 

3.  Considering  courses  for  only  one  version  of  the  aircraft  for  those  aircrew 
positions  for  which  separate  courses  exist  for  different  aircraft  versions.  This  decreased 
the  pool  to  2,300  learning  center  objectives. 

The  goal  was  to  identify  a  sample  of  learning  objectives,  building  from  basic 
knowledge  provided  in  the  learning  center  lessons  to,  but  not  including,  " hands-on "  skills 
provided  in  the  device  sessions.  A  sample  of  254  learning  objectives  was  selected.  Table 
1  provides  a  breakdown  of  the  objectives  by  course. 

To  ensure  a  representative  sample,  the  objectives  indentified  for  each  course  were 
classified  according  to  behavior  and  content  using  the  classification  system  in  the  P-3 
Subject  Matter  Expert  (SME)  Training  Manual.  This  system  classified  the  behavior  as 
remember  or  use  and  the  content  as  fact,  concept,  procedure,  or  rule.  The  definitions  for 
behavior  and  content  in  the  Instructional  Quality  Inventory  (Wulfeck,  Ellis,  Richards, 
Wood,  &  Merrill,  1978)  facilitated  classification  when  added  to  the  P-3  SME  matrix. 
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Summary  of  Objectives 


Course 

Toted  Learning 

Center  Objectives 

Number  in  Sample 

B  MOD/C  Pilot 

775 

64 

B  MOD/C  FE 

225 

29 

B  NFO 

600 

84 

C  SS  1/2 

425 

43 

CSS  3 

275 

34 

Total 

2,300 

254 

Table  2,  which  summarizes  the  classification  of  the  254  objectives  included  in  the 
sample,  shows  that  235  of  them  were  classified  as  "remember"  on  the  behavior  dimension. 
This  is  because  only  learning  center  objectives  were  considered.  Classification  of 
objectives  in  the  device  sessions  would  result  in  large  numbers  in  the  "use"  category.  Of 
the  235  "remember"  objectives,  223  were  classified  as  facts  or  procedures  on  the  content 
dimension.  This  is  representative  of  the  P-3  curriculum  across  aircrew  positions. 

Appendix  A  presents  the  sample  of  learning  objectives  by  aircrew  position.  It 
includes  a  brief  description  of  the  course,  a  summary  of  the  classification  of  the 
objectives,  and  a  detailed  listing  of  the  sample  objectives  for  each  of  the  five  aircrew 
positions. 

Media  Selection  Methodologies 

Technical  reports  describing  and  evaluating  media  selection  techniques  used  in 
military  training  programs  were  reviewed.  Appendix  B  provides  a  description  of  three 
alternative  methodologies  for  selecting  instructional  media.  Two  of  the  media  selection 
techniques  were  considered  representative  of  the  state-of-the-art.  The  TECEP  technique 
described  earlier  is  a  good  example  of  an  algorithmic  selection  model.  Another  example 
is  the  computer-based  Automated  Instructional  Media  Selection  (AIMS)  model,  which  uses 
a  logic  similar  to  TECEP.  The  third,  the  "DELPHI"  method,  was  included  by  direction 
from  NASC.  This  method  was  of  specific  interest  as  a  technique  that  might  prove  to  be 
both  efficient  and  valid  for  selecting  media  in  naval  air  technical  programs  and  that 
NASC  believed  possessed  the  potential  for  minimizing  subtle  biases. 

Methodology  Evaluation 

After  review  of  the  three  media  selection  methodologies,  NASC  selected  the 
"DELPHI"  method  for  evaluation.  Although  the  methodology  evaluated  during  the  study 
was  called  "DELPHI,"  it  had  a  number  of  characteristics  that  distinguished  it  from  the 
Delphi  technique  described  in  the  literature. 

The  classical  Delphi  technique  is  a  set  of  procedures  for  achieving  a  consensus  of 
opinion  from  a  panel  of  experts  for  the  topics  under  consideration.  The  methodology  is 
usually  based  on  a  series  of  mailed  questionnaires  with  information  and  opinion  feedback 


Table  2 


P-3  Classification  Matrix  of  All  Objectives  Included  in  the  Sample 


Content 

Fact 

Concept 

Procedure 

Rule 

Total 

Type  of 

Behavior 

Recall  or 

recognize 

names, 

parts, 

dates, 

places , 

etc. 

Remember 
character¬ 
istics,  or 
classify 
objects, 
events,  or 
ideas 
according 
to  charac¬ 
teristics 

Sequence  of 
steps  remem¬ 
bered  or 
used  in  a 
single  situ¬ 
ation  or  on 
a  single 
piece  of 
equipment 

Remember  or 
use  a 

sequence  of 
steps  that 
apply  across 
situations 
or  across 
equipments 

REMEMBER-re- 
call  or  re¬ 
cognize 
facts,  con¬ 
cept  defi¬ 
nitions, 
steps  of 
procedures 
or  rules 

144 

10 

79 

2 

235 

USE-tasks 
that  re¬ 
quire  clas¬ 
sifying, 
performing 
a  procedure, 
using  a  rule 
with  job 
aids  avail¬ 
able  or 
with  no  aids 
except 
memory 

_0 

_3 

16 

19 

Total 

144 

10 

82 

18 

254 

from  previous  rounds  provided  in  successive  iterations.  As  stated  by  Keller  and  Koen 
(1976),  anonymity  is  of  primary  concern.  The  procedures  are  designed  to  minimize  (1)  the 
effect  of  a  dominant  (but  not  necessarily  more  expert)  individual,  (2)  the  "bandwagon" 
effect,  (3)  the  hesitancy  to  abandon  a  publicly  expressed  position  once  it  has  been  refuted, 
and  (4)  the  pressure  that  exists  to  renounce  an  unpopular  position.  The  procedures  offer 
the  advantage  of  providing  the  expert  with  the  opportunity  to  examine  the  opinions  of 
other  experts  in  his  or  her  own  time  and  benefit  from  the  exposure  to  opinions  of  experts 
with  different  orientations  and  biases. 
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As  described  in  Keller  and  Koen  (1976),  the  traditional  Delphi  technique  appears  to  be 
much  too  time-consuming  for  most  instructional  systems  development  tasks,  including 
media  selection.  An  alternative  method,  the  E-T-E  (estimate-talk-estimate)  method 
(Armstrong,  1978)  would  be  more  appropriate  for  media  selection.  It  has  some  of  the 
same  characteristics  of  the  Delphi  technique  but  is  simpler  and  less  costly  to  apply. 
Further  discussion  on  the  E-T-E  method  and  on  the  traditional  Delphi  technique,  in 
constrast  to  the  "DELPHI"  method  evaluated  in  this  study,  is  provided  in  Appendix  B. 

The  "DELPHI"  method  selected  by  NASC  can  be  described  as  a  relatively 
nonstructured  rating  of  individual  learning  objectives  by  a  panel  of  media  selection 
experts.  The  experts  examine  objectives  one  by  one  and,  during  open  discussion  of  each 
objective,  they  are  expected  to  reach  agreement  concerning  the  medium  of  instruction 
best  suited  for  teaching  the  objective. 

The  goal  of  the  "DELPHI"  used  in  this  study  was  to  obtain  a  consensus  of  expert 
opinion  in  selecting  media,  based  on  effectiveness  only,  for  the  set  of  P-3  training 
objectives.  The  approach  involved  a  team  of  four  experts  in  media  selection.  All  were 
senior  behavioral  scientists  with  10  or  more  years  of  experience  in  human  factors  and 
training  systems  and  extensive  military  training  backgrounds.  Each  has  a  Ph.D.  in 
experimental  or  instructional  psychology.  Two  of  the  members  had  worked  with  the  P-3 
training  program  previously  and  one,  in  fact,  was  involved  in  the  development  of  the 
aircrew  training  program.  The  level  of  expertise  in  particular  media  varied  among 
participants,  with  some  having  particular  expertise  in  simulators,  programmed  instruction 
text,  and  computer-assisted  instruction. 

The  team  was  provided  with  severed  items  to  structure  the  selection  process, 
including: 

1.  The  list  of  candidate  media  specified  by  NASC  in  Task  1. 

2.  The  sample  of  P-3  training  objectives  approved  by  NASC  during  Task  2. 

3.  A  list  of  media  attributes  provided  by  NASC. 

4.  A  brief,  general  description  of  the  process  to  be  followed  provided  by  NASC. 

As  described  earlier,  the  learning  objectives  from  the  P-3  training  program  were 
classified  by  learning  type  prior  to  the  session  and  then  presented  to  the  team  for  use. 
The  objectives  were  drawn  from  the  pilot,  NFO,  and  SS  1/2  courses  in  the  sample  in 
Appendix  A.  The  selected  objectives  matched  the  candidate  media  and  excluded  "hands 
on"  learning  tasks  that  require  simulators,  other  devices  of  high  task  fidelity,  or  any 
actual  equipment  such  as  the  aircraft.  This  selection  emphasized  the  study  of  media 
effectiveness  techniques  for  learning  center  and  classroom  technologies. 

The  list  of  media  attributes  included  color,  text,  motion,  clarity,  flexibility, 
audio/sound,  maintainability,  reliability,  transportability,  applicability  across  objectives, 
operability,  quality  of  responses,  and  managability.  Because  the  media  selection  process 
used  in  the  study  evolved  as  the  study  progressed,  a  more  complete  description  of  it  is 
included  in  the  results. 


RESULTS 

The  results  of  the  media  selection  exercise  indicated  that  the  team  of  experts 
evaluating  the  "DELPHI"  method  lacked  sufficiently  structured  procedures  to  be 
successful  and  efficient  in  selecting  media.  They  required  a  better  definition  of  ground 
rules  and  assumptions  than  was  provided  by  the  "DELPHI"  method  to  solve  the  problems 
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able  or  with  no  aids 
except  memory 


Unit  6--Safety/Survival  Equipment/Emergencies  and  Preflight  Inspections 
Unit  7— Navigation/Mi ssion  Planning 

C.  Unit  8— Oceanography/Submarine  Signatures 
Unit  9— ASW  Procedures  and  Tactics 

Unit  10— LOFAR/DIFAR  Techniques 
Unit  11— Radar/ESM/IRDS/MAD 
Unit  12— Active  Tactics 
Unit  13— Ordnance 
Unit  14 — TACCO  Station 
Unit  15— Communications  Systems 

D.  Unit  16— Tactical  Flight  Scenario 

Since  Phase  C  contains  mostly  classified  material,  the  analysis  was 
confined  to  Phases  A  and  B.  Interviews  with  pilot  instructors  revealed  that 
the  problem  areas  in  the  curriculum  were  concentrated  in  Phase  A,  particularly 
Unit  1.  The  instructors  indicated  that  the  pilot  requires  more  knowledge  of 
how  systems  function  in  order  to  be  able  to  respond  to  multiple  emergencies 
or  other  conditions  where  routine  procedures  may  not  apply.  On  this  basis. 

Lesson  14,  which  covers  the  fuel  supply  system,  and  Lessons  15  and  17,  which 
deal  with  the  propeller  system  components /normal  operations  and  feathering/ 
unfeathering,  were  included  in  the  representative  sample.  Lesson  26,  which 
covers  operation  of  the  PB-20N  Autopilot  system,  was  also  included  since  it 
involves  a  mix  of  factual  and  procedural  objectives  which  are  representative 
of  the  pilot  course. 
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media.  Efforts  made  by  TAEG,  NAVPERSRANDCEN,  and  other  research  organizations 
are  relevant  sources  for  such  data. 

It  should  be  noted,  however,  that  much  of  the  research  on  media  in  the  past  has 
confused  the  use  of  the  media— that  is,  the  instructional  technology  that  can  be  utilized 
on  the  media— with  the  media  hardware.  It  is  of  major  importance  to  define  the 
attributes  of  the  medium  that  make  it  different  enough  to  warrant  selection  of  that 
medium  over  another.  This  is  the  technology  versus  media  hardware  distiction  (Salomon, 
1977).  Research  comparing  television  to  lecture,  CAI  to  textbook,  or  other  media  to  each 
other  is  probably  not  meaningful  and  is,  at  best,  difficult  to  interpret  within  the  context 
of  military  training.  Certainly,  media  selection  methodology  has  not  been  helped  by  this 
type  of  research. 

As  exhibited  during  the  sessions,  the  important  questions  surfacing  are  (1)  what 
features  does  this  median  have  that  we  can  use  and  (2)  under  what  conditions  are  those 
features  more  effective?  These  are  the  basic  data  of  media  selection  and,  unfortunately, 
they  are  usually  missing  when  media  selection  is  carried  out.  In  the  absence  of  such  data, 
one  is  forced  to  define  assumptions  and  ground  rules,  and  formalize  biases  as  explicitly  as 
possible.  This  is  not  to  say  that  all  of  the  algorithmic  media  selection  procedures  being 
used  are  totally  incorrect.  They  are,  in  fact,  the  best  that  can  be  done  at  the  present 
time.  The  use  of  such  procedures  should  not  be  discontinued.  However,  their  weaknesses 
should  be  recognized  as  such. 


RECOMMENDATIONS 

Until  critical  decision  data  are  available  from  additional  media  selection  research, 
any  media  selection  technique  used  must  include  the  rules  and  assumptions  to  be  employed 
if  the  results  are  to  be  interpreted  meaningfully. 


Table  6 


Ratings  of  Seven  Instructional  Objectives  with  Respect  to  the 
NASC-Provided  Attributes,  Using  a  Six-Point  Scale 


Segment 

Criteria 

Pilot 

1/14/1 

Pilot 

1/26/4 

“47576" 

C  SS  1/2 

4/4/7  4/4/8 

8/5/3 

B  NFO 
2/8/13 

Visual  Representation: 

Pictorial 

1 

0 

0 

0 

0 

0 

0 

Graphic 

5 

4 

5 

5 

5 

5 

3 

Color 

1 

1 

0 

0 

0 

0 

0 

Text 

5 

5 

5 

0 

0 

5 

5 

Motion 

2 

0 

0 

0 

0 

0 

Clarity 

0 

0 

0 

0 

0 

0 

Format  Flexibility 

0 

0 

0 

0 

0 

0 

0 

Sound  (Voice) 

3 

3 

3 

0 

0 

3 

3 

Response: 

Verbalize 

5 

5 

5 

0 

0 

0 

5 

Select 

0 

0 

0 

0 

0 

0 

Perform 

0 

0 

5 

5 

5 

0 

Feedback: 

Immediate 

4 

4 

4 

4 

4 

5 

4 

Delayed 

1 

1 

1 

0 

0 

1 

Feedback  Context: 

Correct 

Answer  Only 

5 

5 

5 

0 

0 

5 

5 

Elaborated 

Answer 

0 

0 

0 

5 

5 

0 

0 

CONCLUSIONS  AND  DISCUSSION 

The  obvious  conclusion  regarding  the  media  selection  exercise  is  that  much  more 
structure  is  required  to  accomplish  successfully  the  media  selection  task,  as  evidenced  by 
the  attempted  structuring  that  occurred  during  the  session.  However,  it  is  doubtful  that 
it  was  possible  to  develop  a  set  of  assumptions  and  ground  rules  that  would  have  been 
acceptable  to  all  members  of  the  team.  There  would  have  been  disagreements  with  some 
of  the  definitions  even  if  much  more  time  had  been  available.  The  remaining  disagree¬ 
ments  were  a  function  of  the  team  members'  particular  experiences  and  resulting  biases. 

The  lack  of  consensus  among  team  members  and  their  constant  questioning  of  each 
other  is  not  so  much  a  reflection  on  the  capabilities  of  the  participants  as  it  is  on  the 
state  of  knowledge  of  differential  media  effectiveness  and  media  selection  itself. 
Continued  efforts  should  be  made  to  compile  data  on  both  media  hardware  and  the  use  of 


Table  4 


Six-Point  Scaling  Systems  Developed  to 
Assist  in  Judging  Media  Attributes 


System 

Scale  Alternatives 

Applicability  of  Media  Attributes 

0  -  Not  Applicable 

1  -  Of  Minimum  Help 

2  -  Of  Some  Help 

3  -  Moderately  Helpful 

4  -  Very  Helpful 

5  -  Essential 

Capability  of  Media 

0  -  None 

1  -  Minimal  Capability 

2  -  Some  Capability 

3  -  Substantial  Capability 

4  -  Some  Limits 

5  -  Complete 

The  procedure  that  evolved  was  to  rate  each  of  the  listed  media  on  one  of  the  six- 
point  scales  with  respect  to  its  capability  to  handle  each  of  the  media  attributes  provided 
by  NASC.  These  ratings  are  shown  in  Table  5.  Following  this,  an  attempt  was  made  to 
rate  each  of  the  chosen  instructional  objectives,  using  a  six-point  scale,  with  respect  to 
the  importance  of  the  media  attributes.  Although  Table  6  shows  the  results  of  the  ratings 
for  the  seven  objectives  completed,  there  were  numerous  examples  of  disagreement  that 
are  not  shown  in  the  table.  Although  the  intent  was  to  match  the  ratings  of  available 
media  with  those  of  the  sample  objectives,  it  became  evident  that  the  evolved  procedures 
had  shortcomings  and  that  the  desired  results  were  not  being  achieved  within  the  time 
available  for  the  process.  Because  of  this,  and  as  a  result  of  the  disagreement  about  the 
results  shown  in  Tables  5  and  6,  the  process  was  terminated. 
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The  list  of  candidate  media  was  determined  to  be  insufficient  in  detail.  While  the 
team  members  were  willing  to  work  with  this  particular  media  pool,  the  lack  of 
information  about,  or  definition  of,  the  *  dia  was  troublesome.  For  example,  a  PLATO 
CAI  system  may  have  several  configurations  with  different  capabilities.  Since  the 
configuration  was  not  defined,  assumptions  had  to  be  made  as  to  which  capabilities  would 
exist.  Much  additional  discussion  ^entered  on  the  design  technology  of  the  mediated 
instruction  as  opposed  to  the  media  l«ardware.  That  is,  does  one  assume  that  a  lecture 
would  be  well  designed  and  instructor  implemented,  provide  interaction  with  students, 
elicit  questions,  provide  feedback,  provide  demonstrations,  allow  for  individual 
differences,  provide  feedback  to  the  instructor  on  student  progress,  or  be  supported  by 
quality  learning  materials?  All  of  these  are  certainly  possible  with  a  lecture,  but  it 
became  necessary  for  the  media  selection  team  to  define  such  items  and  make 
assumptions.  Biases  about  the  capability  of  technology  or  quality  of  the  media  also 
became  apparent  during  these  discussions.  Some  participants  believed  that  high  quality 
could  be  obtained  with  one  media  form  and  not  with  another  or,  at  least,  that  more 
difficulty  in  standardization  of  quality  might  be  inherent  in  the  media.  Such  expert 
opinion  is,  of  course,  actually  biased.  As  the  biases  toward  technology  and  media 
surfaced,  the  discussions  became  more  intense,  requiring  more  definitions  and  assump¬ 
tions,  and,  to  some  extent,  compromising  of  individual  positions.  Although  members 
reached  agreement  in  some  areas,  they  never  achieved  total  consensus. 

The  learning  objectives  were  also  subject  to  controversy.  Team  members  raised 
questions  concerning  the  completeness  of  the  objectives,  since  some  did  not  have 
satisfactory  standards  or  sufficient  descriptions  of  conditions.  They  constantly  referred 
questions  to  the  "P-3  expert"  concerning  the  requirements  of  learning  tasks  in  accomplish¬ 
ing  the  objective.  The  grouping  of  objectives  by  learning  classification  also  posed  a 
problem  for  two  of  the  members  who  were  unfamiliar  with  the  system  utilized.  Although 
this  system  is  part  of  the  Instructional  Quality  Inventory  (IQI)  (Wulfeck  et  al.,  1978)  and  is 
utilized  in  several  naval  air  training  programs,  agreement  as  to  the  specific  classification 
of  each  objective  was  sometimes  difficult.  In  addition,  the  implications  of  the 
classification  for  media  selection  were  often  not  evident. 

The  media  attributes  also  caused  concern.  During  the  course  of  the  session,  the  list 
of  attributes  was  restructured  and  several  attributes  were  redefined  to  achieve  consensus 
among  team  members.  They  added  such  items  as  feedback  (with  various  explicit  forms), 
graphics,  and  specific  response  (verbalize,  select,  perform)  to  the  list.  The  team  members 
also  decided  that  voice  and  sound  should  be  two  separate  attributes  for  evaluation.  In 
addition,  flexibility  was  redefined  to  mean  flexibility  in  format  as  to  how  the  courseware 
could  be  structured  on  that  media.  Some  items,  such  as  reliability  and  maintainability, 
were  questioned  as  being  appropriate  for  an  evaluation  of  training  effectiveness. 
Furthermore,  judgment  was  difficult  to  make  as  to  the  reliability  or  maintainability  of 
particular  media.  Although  participants  had  explicit  experience  with  some  of  the  media 
forms,  it  was  felt  that  there  is  not  enough  unbiased  data  with  which  to  judge  many  media 
candidates. 

During  the  session,  team  members  discovered  that  they  could  judge  media  attributes 
only  on  the  basis  of  some  form  of  scaling  system.  Therefore,  scaling  systems  were 
developed.  Several  six-point  scales  evolved  because  a  particular  scale  did  not  always 
seem  to  fit  ail  attributes.  Two  such  scales  are  shown  in  Table  4. 
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inherent  in  the  media  selection  process.  During  the  course  of  the  9-hour  session,  a 
number  of  discussions  were  held  that  revealed  a  deep  divergence  of  views  among  members 
of  the  group,  all  of  whom  are  respected,  experienced  professionals  in  the  field.  The  team 
evaluated  only  seven  objectives  for  media  selection  during  the  session  and  never  reached 
total  agreement  even  on  those  seven.  The  seven  objectives  and  their  characteristics  are 
shown  in  Table  3.  The  following  paragraphs  describe  some  of  the  major  problems 
encountered. 


Table  3 

Characteristics  of  Learning  Objectives  Evaluated 


Unit/  Classification 


Crew 

Position 

Lesson/ 

Segment 

Training  Segment 

Behavior 

Content 

Pilot 

1/14/1 

State  the  location  of 
(1)  fuel  tanks  one 
through  four  and  (2) 
fuel  tank  five. 

Remember 

Fact 

Pilot 

1/26/4 

List  the  four  procedural 
steps  for  engaging  the 

AFCS. 

Remember 

Procedure 

C  SSI/ 2 

4/4/6 

State  the  four  distinctive 
signature  characteristics 
for  the  Soviet  type  1 
diesel  submarine. 

Remember 

Concept 

C  SS 1/2 

4/4/7 

Given  multipoint  dividers, 
frequency  scales  and 
lofargrams  with 
calibration  marks, 
classify  signatures  of 

Soviet  type  1 
diesel  submarines. 

Use 

Concept 

NBFO 

2/8/13 

Given  an  LTN-72  INS  data 
sheet  and  the  necessary 
information,  complete 
the  form. 

Use 

Procedure 

C  SSI/2 

4/4/8 

Given  the  HP-67  calculator, 
the  Fleet  Mission  Program 
Library  VP  Inflight  Manual, 
and  VP  program  cards, 
analyze  signatures  for 

Soviet  type  1  diesel 
submarines. 

Use 

Rule 

C  SSI/2 

8/5/3 

List  the  four  steps  to 
determine  layer  depth 
and  sea  surface 
temperature. 

Remember 

Rule 

Sample  of  Training  Objectives 
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The  B  MOD/C  Flight  Engineer  (FE)  course  consists  of  the  following  units: 
Unit  1— Aircraft  Familiarization 
Unit  2--0peration  of  PT  Computer 
Unit  3— Aircraft  Performance 
Unit  4— Normal  Procedures 
Unit  5— Malfunction  and  Emergency  Procedures 
Unit  6— Weight  and  Balance 
Unit  7— Aircraft  Servicing 
Unit  8— Aircraft  Inspection 
Unit  9— Survival  Equipment 
Unit  10— Flights 

Unit  11— Flight  Procedures  and  Planning 

Unit  12— TAC  ana  Nav  Extended  Flights 

The  lessons  chosen  from  Unit  5,  emergency  shutdown  procedures,  engine 
malfunctions,  and  propeller  malfunctions,  exhibit  a  mix  of  factual  and  procedural 
objectives  which  are  representative  of  the  FE  course. 
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Objectives  from  Naval  Flight  Officer  Course 

The  Naval  Flight  Officer  (NFO)  course  consists  of  the  following  units 
organized  by  phases  in  the  Master  Course  Syllabus: 

A.  Unit  1--Aircraft  Familiarization 

Unit  2— Equipment— Hardware  and  Operation 

Unit  3— Flight  Procedures/Weather/Maintenance  Logs 

B.  Unit  4— Survival /Emergency  Equipment  and  Procedures 
Unit  5— NAV-1  FI i ght/NATOPS  Review 

Unit  6— Oceanography/Submarine  Signatures 
Unit  7— TAC  Hardware  and  Procedures 
Unit  8— Tactics/DRT/LOFAR 

C.  Unit  9— DIFAR/LOFAR  Procedures  and  Techniques 
Unit  10 — Radar/ESM/MAD 

Unit  ll""Active  Tactics 
Unit  12— Ordnance 

Unit  13— 2F69  Trainer  Device  Sessions 
Unit  15— Tactics  and  Tactical  Communication 

D.  Unit  16— 2F69  Device  Sessions/Emergencies 
Unit  17— Submarine  Signatures 

The  objectives  selected  for  the  BNFO  course  come  from  two  lessons,  the 
LTN-72  Inertial  Navigation  System  (Unit  2/Lesson  8)  and  ASN-124  Tactical  Hardware 
(Unit  7/Lesson  3).  Both  the  LTN-72  and  the  ASN-124  lessons  involve  memorization 
of  complex  pieces  of  equipment  and  associated  procedures.  Basically  this 
involves  memorization  of  tables  detailing  components  and  their  functions,  power 
sources,  circuit  breakers,  and  locations;  the  effects  of  different  switch  settings, 
buttons,  keyboard  inputs,  etc;  the  meaning  of  display  mnemonics  and  symbology; 
signal  flow;  and  procedural  steps  involved  in  using  the  equipment. 

These  lessons  are  representative  of  the  materials  in  Phase  A,  which  empha¬ 
sizes  familiarization  with  equipment  and  procedures  prior  to  device  sessions  in  the 
trainers  and  aircraft.  The  lessons  in  Phases  B  and  C  are  related  to  tactics  and 
Involve  some  classified  materials,  so  they  were  not  included  in  the  sample. 
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Sample  of  Training  Objectives 
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7/3/7  Given  a  diagram  of  the  TOC  tray  and  a  list  of  several  TDC  switches.  Remember  Fact 

state  the  function(s)  of  each. 
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aining  Effectiveness  and  Cost  Effectiveness  Prediction  (TECEP)  Technique 

The  TECEP  technique  (Braby,  Henry,  Parrish,  <Sc  Swope,  1975)  was  developed  by  the 
‘aining  Analysis  and  Evaluation  Group  (TAEG)  to  provide  learning  principles  appropriate 
>  Navy  job  tasks  and  a  method  for  selecting  cost-effective  instructional  delivery  systems 
at  support  the  use  of  these  learning  principles.  Aagard  and  Braby  (1976)  state  that  the 
E.CEP  technique  uses  learning  guidelines,  based  in  part  on  those  used  by  Willis  and 
eterson  (1961)  and  Gagne  (1965),  and  learning  algorithms  developed  to  combine  and 
tquence  the  guidelines.  As  defined  by  Braby  et  al.  (1975),  "a  learning  algorithm  is  a  step- 
f-step  prescription  for  a  student  to  follow  in  learning  any  specific  task  in  a  class  of 
arning  tasks. .  ." 

The  TECEP  technique  was  designed  as  a  three-step  job  aid  to  be  used  by  training 
rstem  designers  in  the  media  selection  process.  Figure  B-l,  taken  from  TAEG  Report 
o.  16,  provides  a  detailed  picture  of  the  three  steps  in  the  process.  However,  only  Steps 
and  2  are  described  herein  since  Step  3,  cost  considerations,  was  beyond  the  scope  of 
us  study.  A  discussion  of  modifications  to  enhance  NAVAIR  use  of  the  TECEP  technique 
also  provided. 

Step  1 

The  initial  step  is  to  classify  and  group  the  training  objectives  according  to  the  type 
f  learning  algorithm  required  to  accomplish  the  objective.  This  involves  matching  each 
-aining  objective  in  a  proposed  training  system  with  the  name  of  the  learning  algorithm 
ppropriate  for  achieving  the  objective.  The  learning  guidelines  and  associated  algorithms 
re  defined  as  follows: 

1.  Recalling  bodies  of  knowledge. 

2.  Using  verbal  information. 

3.  Rule  learning  and  using. 

4.  Decision  making. 

5.  Detecting. 

6.  Classifying. 

7.  Identifying  symbols. 

8.  Voice  communicating. 

9.  Recalling  procedures  and  positioning  movement. 

10.  Steering  and  guiding,  continuous  movement. 

11.  Performing  gross  motor  skills. 

12.  Attitude  learning. 

As  shown  in  Figure  B-l,  Step  1  requires  using  a  table  describing  the  12  types  of 
•arning  algorithms  and  the  characteristics  of  the  training  objectives  that  they  support, 
igure  B-2,  taken  from  TAEG  Report  No.  16,  is  an  example  of  the  table  used.  The 
haracteristics  of  the  training  objective  are  compared  with  the  action  verbs,  behavioral 
ttributes,  and  examples  of  objectives  that  can  be  achieved  with  the  algorithm  to  verify 
lie  classification.  Training  objectives  that  are  classified  alike  are  grouped  into  sets  for 
nalysis  in  Step  2. 

Step  2 

The  second  step  is  to  identify  the  instructional  delivery  systems  that  will  support  the 
se  of  the  learning  algorithm  required  for  each  set  of  training  objectives.  As  shown  in 
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Table  A-10  (Continued) 
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Objectives  from  Sensor  Station  Operator  3  Course 

The  Sensor  Station  Operator  3  (C.SS  3)  course  consists  of  the  following 
units  organized  by  phases  in  the  Master  Course  Syllabus: 

B.  Unit  1— Aircraft  Familiarization 
Unit  2— Data  Processing  System 

l’ Tit  3— Radar/IFF 

C.  Unit  4— Emergency  Equipment 
Unit  5— MAD 

Unit  6—  ESM 

D.  Unit  7— Tactics 

Objectives  from  five  lessons  in  Unit  3,  ICS  components  and  voice  procedures, 
scope  interpretation,  radar  operations— parts  1  and  2,  and  radar  off-line 
operation,  were  selected  as  representative  of  the  tasks  taught  in  the  SS  3  course. 
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Sample  of  Training  Objectives 
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Summary  of  Training  Objectives  Classification 
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able  or  with  no  aids 
except  memory 


The  Sensor  Station  Operator  1/2  (C  SS  1/2)  course  consists  of  the 
following  units  organized  by  phases  in  the  Master  Course  Syllabus: 

A.  Unit  1 — Aircraft  Familiarization 
Unit  2 — AQA-7  DIFAR  System  Controls 

B.  Unit  3--Harmonic  Families 

Unit  4 — Surface  Vessel  Classification 
Unit  5— Associated  Equipment  and  Function 

C.  Unit  6— Classification,  Detection,  Localization 
Unit  7 — Equipment  Malfunctions 

Unit  8— Analysis  and  Monitoring 

D.  Unit  9 — Tactics 


The  representative  sample  of  objectives  for  the  SS  1/2  course  includes 
lessons  from  each  unit  in  Phases  A,  B,  and  C.  Units  2,  5,  and  7  focus  on 
equipment  function,  procedures,  malfunction  identification,  and  preflights 
through  the  use  of  workbooks,  videotapes,  and  aircraft  device  sessions.  .  Units 
3,  4,  and  6  focus  on  gram  analysis  procedures. 
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TABLE  2.  TWELVE  TYPES  OF  LEARNING  ALGORITHMS  WITH  THE 

CHARACTERISTICS  OF  TRAINING  OBJECTIVES  THEY  SUPPORT 
(Si*  TAEG  Report  23  for  actual  algorlthaa) 


NAMES  Or 
LEARNING 
ALGORITHMS 

CHARACTERISTICS  OF  TRAINING  OBJECTIVES  THAT  CAN  BE  ACHIEVED  WITH 
SPECIFIC  ALGORITHMS 

ACTION 

VERBS  BEHAVIORAL  ATTRIBUTES  EXAMPLES 

1.  RECALLING 
BODIES  OF 
KNOWLEDGE 

I 

Answer 

Define 

Expreee 

Infer* 

Select 

1.  Concerns  verbal  or 
symbolic  learning. 

2.  Concerns  acquisition  and 
long-ter*  maintenance  of 
knowledge  eo  that  it  cen 
be  recalled. 

1.  Recalling  equipment  nomen¬ 
clature  or  functions. 

2.  Recalling  system  functions, 
such  ae  the  complex  rela¬ 
tions  between  system  input 
and  output. 

3.  Recalling  physical  laws, 
such  as  Ohm's  lav. 

4.  Recalling  specific  radio 
frequencies  end  other 
discrete  facta. 

2.  USING 

VERBAL 

INFORMA¬ 

TION 

Apply 

Arrange 

Chooaa 

Compare 

Determine 

1.  Concerns  the  practical 
application  of 
information. 

2.  Generally  follows  tha 
initial  laaming  of 
information  through  tha 
use  of  the  guidelines 
for  Recalling: Bodies' 
of  Knowledge. 

3.  Limited  uncertainty  of 
outcome. 

4.  Usually  little  thought 
of  other  alternatives. 

1.  Baaed  on  academic  knowledge, 
determine  which  equipment  to 
use  for  a  specific  real 
world  task. 

2.  Baaed  on  an  academic  knowl¬ 
edge  of  the  system,  compare 
alternative  modes  of  opera¬ 
tion  of  a  piece  of  equipment 
and  datermlne  tha  appropri¬ 
ate  mode  for  a  specific  real 
world  situation. 

3.  Baaad  on  memorised  knowledge 
of  radio  frequencies,  choose 
the  correct  frequency  in  a 
specific  real  world  situa¬ 
tion. 

3.  RULE 
LEARNING 

AND  US INC 

Chooee 

Conclude 

Deduce 

Predict 

Propose 

Select 

Specify 

1.  Choosing  a  eourae  of 
action  based  on  apply¬ 
ing  known  rules. 

2.  Frequently  involves 
"If... Then”  sltuotlone. 

3.  The  rule*  ere  not 
questioned,  the  decision 
focuses  on  whether  the 
correct  rule  is  being 
applied. 

1.  Apply  the  "rules  of  the 
road . " 

2.  Solve  mathematical  equatlono 
(both  chooalng  correct 
equation  and  the  mechanics 
of  solving  the  equation). 

3.  Carrying  out  military 
protocol. 

4.  Selection  of  proper  tire 
extinguisher  for  different 
type  fires. 

5.  Ualng  ccrract  graaar  in 
novel  situations,  covered 
by  rules. 

4.  MAXING 
DECISIONS 

Chooee 

Design 

Diagnoee 

Devtlop 

Evaluate 

Forecast 

Forvulate 

Organist 

Selact 

1.  Choosing  a  course  of 
action  when  alternatives 
are  unspecified  or 
unknown. 

2.  A  successful  course  of 
action  la  hot  readily 
apparent. 

easeful  courses  of 
action  are  not  readily 
apparent. 

4.  The  relative  value  of 
possible  decisions  must 
bt  considered  -  includ¬ 
ing  possible  trade-offs. 

3.  Frequently  involves 
forced  dec is lone  made  in 
a  short  period  of  time 
with  soft  information. 

1.  Choosing  frequencies  to 
search  in  an  ECM  search  plan. 

2.  Choosing  torpedo  settings 
during  a  torpedo  attack. 

3.  Threat  avalisation  and  weapon 
aanlgnmant . 

4.  Choice  of  tactics  In  com¬ 
bat  -  wide  range  of  options. 

3.  Choosing  a  diagnostic 

strategy  In  dealing  with  a 
malfunction  in  a  complex 
piece  of  equipment. 

6.  Choosing  to  abort  or  commit 
oneself  to  land  upon  reach- 
log  the  critical  point  in 
the  glldepath. 

Figure  B-2.  Example  of  TECEP  Learning  Algorithms 


Figure  B-l,  Step  2  consists  of  two  substeps— identification  of  instructional  delivery 
systems  and  practicality  tests— with  two  alternatives  provided  for  accomplishing  the  first 
substep. 

In  the  first  substep,  the  instructional  delivery  systems  with  the  stimulus,  response, 
and  feedback  capabilities  required  to  support  the  events  in  the  learning  algorithm  are 
identified.  Instructional  delivery  system  selection  charts  were  developed  for  each  of  the 
12  learning  algorithms  to  serve  as  job  aids  for  training  system  designers  to  complete  this 
task.  As  shown  in  the  example  in  Figure  B-3,  a  comprehensive  list  of  instructional 
delivery  systems  is  provided  across  the  top  of  the  chart,  while  special  selection  criteria 
are  listed  on  the  side  with  an  "X"  placed  in  the  cells  of  systems  that  meet  the  special 
criteria.  The  training  system  designer  can  easily  identify  candidate  instructional  delivery 
systems  by  specifying  the  relevant  selection  criteria  on  the  left  side  and  looking  for  "Xs" 
in  those  rows. 

An  alternate  approach  to  this  substep  is  provided  to  allow  experienced  training 
system  designers  to  consider  instructional  delivery  systems  not  included  in  the  charts. 
Two  job  aids  are  provided:  (1)  a  list  of  55  generic  media  characteristics  that  includes 
stimulus,  response,  and  feedback  and  (2)  a  list  describing  89  general  types  of  instructional 
media  that  can  be  incorporated  into  instructional  delivery  systems.  Examples  from  each 
list  are  provided  in  Figures  B-4  and  B-5  respectively.  From  these,  the  generic  media 
characteristics  required  to  implement  the  learning  algorithms  can  be  specified  and  the 
media  containing  these  characteristics  can  be  identified.  This  approach  requires  expert 
knowledge  of  media,  the  learning  algorithms,  and  the  subject  matter. 

The  second  substep  is  testing  the  candidate  instructional  delivery  systems  for 
practicality.  Each  delivery  system  should  be  evaluated  in  terms  of  the  following  11 
criteria  so  the  impractical  delivery  system  candidates  are  eliminated  before  the  costing 
exercise  in  Step  3. 

1.  Marginal  technical  solution. 

2.  State-of-the-art  system. 

3.  Size  of  the  system. 

k.  Interface  with  existing  programs. 

5.  Long  lead  time  to  produce  the  system. 

6.  Budget  cycle  constraints. 

7.  User  acceptance  of  innovations. 

8.  Resources  for  courseware  development. 

9.  High  cost  system  alternative. 

10.  Learning  style  of  trainees. 

1 1 .  Any  other  applicable  constraints. 

Enhancements 

The  use  of  training  tasks  and  training  stages  provides  a  method  to  identify  the 
learning  algorithms  required  to  support  the  training  when  training  objectives  are  not 
available  at  the  time  media  selection  decisions  must  be  made.  Figure  B-6,  taken  from 
TAEG  Report  No.  1,  is  an  example  of  this  method.  To  test  the  feasibility  of  the  TECEP 
model,  it  was  applied  to  the  TA-4  aircraft  training  program  (Braby,  Micheli,  Morris,  & 
Okraski,  1972).  As  shown  in  Figure  B-6,  the  stages  of  training  for  each  task  and  specific 
tasks  within  each  stage  of  training  were  matched  with  primary  task  categories,  which 
roughly  correspond  to  the  learning  algorithms  currently  used  in  Step  1. 
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TABLE  3.  INSTRUCTIONAL  DELIVERY  SYSTEM  Q!ART  FOR  THE  ALGORITHM 


Directions  t 

To  choose  a  dalivary 

systemi 

X.  Piece  a  V"  (light 
paneil)  in  boxes 
representing  criteria 
(rows)  that  muat  ba  mat. 

2.  Salact  the  delivery 
aystena  (columns)  that 
Iiavo  am  in  nsrh  rnn 

d«* ig<»« t«d  by  a  V*  • 

Thaaa  ara  tha  candidata 
^delivery  systems. 

Criteris  for 
Selacting 
Znstructionsl 
Delivery 

Systems 

\ 

RECALLING  BODIES  OF  KNOWLEDGE 


AlttrRttlvi  Instructional  Delivery  System 


Delivery  Approaches  Delivery  Approaches  NOT 

Permitting  the  Application  Permitting  Complete 

of  All  Learning  Guidelines  Application  of  Learning 

and  Algorithm  Guidelines  and  Algorithm 


Stimulus  Criteris 

•  Visusl 

Limited 

Full 

•  Visusl 

Spectrum 

Pull  Color 

•  Audio 

Voice  Sound  Range 

Training  Setting  Criteria 
%  Individual  Trainees  at  Pined  Location 
•  Individual  Trainees  with  Simultaneous 


Instruction  at  Many  Location* 

•  Individual  Tralnaaa  with  Independant 
Instruction  at  Any  Location 

•  Small  Croup 

•  Large  Group  et  a  Single  Location 

•  Team  Setting 


-f  H  MX 

x  -  «  ■ 

o  >  o  u 

«  can 

x  *  v  —  • 

O'  X  u  V  O'  H 

O'  c  u  o  a  c 

c  -*  -*  o  s  -  O' 

•*  -C  ^  w  *  X  C 

X  o  Of  u  U  — 

u  c  bh  O'  C 

m  •  o  o  m 


Administrative  Criteria 

•  Si\«  of  Couraaware  and  Special  Hard¬ 
ware  Development 

Local 

Central 

•  magnitude  of  Acquisition  Cost 

Low 


Figure  B-3.  Example  of  TECEP  Instructional  Delivery  System  Selection  Chart 
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tabu  is.  comic  cuAiAcmjsTics  or  tbaikibc  media 


fTDVUIS  CAPABILITIES 

tuwI  r»rw 

X*  tlMil  XlBtmwitrlc  •  wHi,  auab.r.  u4  oth.r 
•jafcol*  ptpmmaS  graphically . 

2.  Tlaual  Hctorl.l.  flana  -  •  nm-dlMsalsuX 
lM|t,  e  ri|raamt«tiM  la  tbs  for*  of  o 
fkoto|raf»i  or  fr«ria|. 

1.  TIpuaI  Lina  Coiwtrmtlwi,  Tlaiw  •  a  t«o- 
Alaaaalaaal  Ugura  aada  of  llaaa,  auch  aa 
o  anrheastlcal  cum  or  iraph. 

4.  Visual  Obloct.  Solid  -  •  three-dlaenaional  1m|* 
or  reality  chat  la  vlaw4  fro*  os c trior 
perspectives. 

i 

3.  Visual  tnvlrowant  -  A  throe-dlaenslonal  i*a|o 
or  reality  that  io  viewed  fro*  Isolds. 

n»— i  wwwn 

A-  Tlwwl  Belli  -  a  atatlc  Tlaual  (tall,  aa  vlth  a 
atill  photograph,  drawing ,  or  prlatad  paga. 

1.  Tlaual  Lloltad  WoTouaat  -  a  haalcal)?  atatlc 
oiaual  f laid  ulth  alaaaota  that  cap  pa  nada  to 
noro,  aa  with  aa  aulaatad  traaapargpgjr  or  a  lop  la" 
paaol  with  awltchaa  taat  uoto. 

(•  Tlaual  Full  Hot— ant  .  a  Tlaual  (laid  la  which 
all  alaaoeta  caa  oowa ,  aa  with  a  uotlon  plctura, 
(light  alaulatar,  or  oparatloual  aircraft. 

».  Tlaual  CtcIIc  Hn.nmt  -  a  Tlaual  (laid  which 
■oral  through  a  (load  aaguanca  aad  than  rapoata 
tha  aaguaaca  la  a  repetitive  uarmat ,  aa  with 
a  Hid  loop. 

rtmi 

10.  Heck  end  Whitt  -  o  visual  fltld  coopottd  of 
t liber  black  or  white  tla*anttt  at  with  tha 
printed  past  or  lino  drawing*. 

11.  Cray  Scale  -  a  visual  field  competed  of  black, 
whlta  and  continuous  gradations  of  |r«y.  aa 
with  a  black  and  white  photograph  or 
television  picture. 

12.  Color  -  a  visual  field  composed  of  various 
esgaeats  of  tbs  visual  apectrua,  aa  with 
color  television  or  action  pictures. 

feels 

13.  Exact  feels  -  actual  visual  field  or  a  one-to- 
one  replication  of  thet  field  so  with  a  full- 
sited  »o«k-up.  aiaulator,  or  operational 

eye tea. 

14.  f report lehel  fctlt  -  e  representation  of 
reality  la  other  then  full  scale ,  ouch  as  a 
seeled  aodel  aep  or  photograph. 

Audi# 

U.  Tplcc  Bound  »on«.  -  a  Halt*  guallcp  e( 
eeuad  which  eaeblea  specie  wards  to  be 
wood  aa  tha  aodlwa  af  ewa  test  lone,  hut 
act  awitad  to  aora  d sated lag  tatkt,  auch 
as  autlc  or  aouad  rscegnltloa  exercises. 


If*  Full  Sound  fangs  -  a  quality  of  sound  reproduction 
that  coatalaa  all  tha  significant  eleatntt  of  the 
aound  aad  it  suited  to  the  dsnandlng  teek  of 
sound  recognition  exorcises. 

17.  Aablant  Sounds  -  a  eoaplax  aound  envlronaent  with 
aounde  eaenetlng  froa  various  source*  end  froa 
various  directions.  Including  background  noise 
and  task  significant  sounds. 

Other 

If*  Tactile  Cues  -  signal*  received  through  the  sense 
of  touch,  Including  sensations  related  to  texture, 
•1st  or  shape. 

If.  Internal  Stimulus  Motion  Cues  -  the  sensations 
felt  by  a  person  whan  he  aovee  hie  era,  leg, 
fingers,  etc. 

•  20.  External  Stlaulue  Motion  Cues  -  the  eeneatlone  felt 
by  a  parson  whan  ha  le  aovad  by  aoae  outside  force 
In  such  a  way  that  hie  body  experiences  roll, 
pitch,  yaw,  heave,  away  and/or  surge. 

TRAINEE  RESPONSE  MODES 

21.  Covert  Response  -  e  response  which  the  trelnae 

creates  in  his  mind  but  does  not  express  la  an 
ebservable  Banner.  •  , 

22.  Multiple  Choice  -  e  response  aode  In  which  a 
trainee  selects  e  reeponee  froa  a  Halted  eat  of 
reeponees . 

23.  Pre-progreweaed  Verbal  Performance  -  a  response 
aode  in  which  e  trelnee  creecee  e  short  answer 
to  a  question  having  s  Halted  sat  of  correct 

answers . 

24.  Free-Style  Written  Performance  -  a  reeponee  aode 
in  which  s  trainee  writes  a  roaponse  in  hie  own 
words. 

25.  Decision  Indicator  -  a  verbal  or  perceptual 
actor  reaponse  In  which  the  trainee  indicate* 
that  he  hat  made  a  divergent  type  decision. 

26.  Voice  Performance  -  a  response  aode  in  which  s 
trainee  speaks,  including  conversation. 

27.  Pine  Movement  Manipulative  Acts  -  s  response 
aode  In  which  e  trainee  aakee  discrete  and 
smell  movements  of  dials,  swltchts,  keys  or 
aakss  sansltivs  adjustments  to  instruments. 

Act  asy  lnvolvs  use  of  sasll  instruments. 

28.  Broad  Movement  Manipulative  Acts  -  s  response 
aode  lo  which  s  trainee  askes  large  movements 
of  lavtrs  or  whesls  on  lsrgt  pieces  of  equlp- 
asnt  or  by  tha  use  of  hand  held  tools. 

29.  Trscklnt  -  a  rssponss  aode  in  which  a  trainee 
continuously  controle  a  constantly  changing 
system,  such  as  steering  an  automobile  or 
holding  a  compass  bearing  in  steering  a  ship. 

30.  Procedural  Manipulative  Acts  -  s  rssponss  mods 
la  which  a  trains#  performs  the  sequence  of 
etspe  In  a  procedure,  ouch  as  in  the  carrying 
out  of  the  lteas  on  the  checklist  for  pre- 

f lighting  en  slrcreft  or  tuning  on  e  reder 
eystea. 


Figure  B-4. 


Example  of  TECEP  Generic  Media  Characteristics 
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TABLE  16.  MEDIA  POOL 


PRINT  MATERIALS 

CASE  STUDY  FOLDER  -  A  folder  of  detailed  background  information  on  a 
problem  requiring  a  decision  or  plan  of  action;  to  be  read  by  the 
trainee  prior  to  his  (1)  making  a  decision  on  how  to  resolve  the 
issue  and  (2)  participating  in  a  critique  on  various  solutions. 
Various  forms  of  folders  are  used  in  support  of  such  methods  of 
instruction  as  the  Case  Study,  Incident  and  In-Basket  methods  of 
management  and  leadership  trairving.~ 

FLASH  CARDS  -  A  se\  of  cards  designed  to  be  used  by  an  instructor  in 
front  of  a  group  of  trainees  to  drill  the  group  in  the  recall  of 
memory  type  information. 

PRINTED  MATERIALS  -  HANDOUTS  -  Handouts  are  a  class  of  printed  materials 
issued  to  a  student  for  his  use  and  retention  to  augment  regular 
instructional  materials.  ..They. are  usually  instructor  prepared, 
machine  copied  materials  of  ond  or  two  pages  highlighting  specific 
topics  or  updating  existing  materials. 

PRINTED  MATERIALS  -  PERFORMANCE  AIDS  -  Performance  aids  are  a  class  of 
printed  materials  that  aid  in  job  performance  by  providing  data 
that  should  not  be  ponmitted  to  memory.  They  include  checklist 
routines,  conversion  tables,  equipment  test  tolerance  matrices  and 
the  like. 

PRINTED  MATERIALS  -  REFERENCE  BOOKS  -  Reference  books  are  a  class  of. 
printed  materials  used  to  identify  certain  facts  or  for  background 
information  such  as  dictionaries,  encyclopedias  or  technical 
publications. 

PRINTED  MATERIALS  -  REFERENCE  CHARTS  -  Reference  charts  are  a  class  of 
printed  material  pictorial ly  displaying  data  used  to  identify 
certain  facts  or  for  background  information.  Included  are  data 
charts,  schematic  diagrams,  topographical  maps  and  the  like. 

PRINTED  MATERIALS  -  SELF-SCORING  EXERCISES  -  Self-scoring  materials 
include  exercises  and  quizzes  used  in  conjunction  with  standard 
curriculum,  or  programed  instruction.  The  class  includes  electro- 
graphic  or  mark  sense  materials  scored  by  keys  or  computer,  punch 
mark  and  other  mechanical  score  indicating  equipments,  chemically 
scored  materials,  etc.,  that  have  the  capability  of  providing  near 
iranediate  student  feedback  without  the  use  of  prolonged  scoring 
procedures. 


Figure  B-5.  Example  of  TECEP  Media  Pool 
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Table  2 

Categorization  of  Tasks  in  the  Familiarization  Stage  in  the 
TA-4J  Advanced  Jet  Navy  Undergraduate  Pilot  Training  Program. 


Stage 

Tasks  Within  Stage 

Primary  Task  Categories 

Familiarization 

Cockpit  Procedures 

Recalling  Procedures 

Non-Verbal  Identification 

Positioning  &  Serial  Movement 

i 

Preflight  Inspection 

Recalling  Procedures 

Non-Verbal  Identification 

Making  Decisions 

Normal  Operating 
Procedures  .  . .  ,  , 

Recalling  Procedures 

Non-Verbal  Identification 

Continuous  Movement 

Positioning  and  Serial  Movement 

Nosewheel  Steering  and 
Taxi  Procedures. 

Recalling  Procedures 

Non-Verbal  Identification. 

Continuous  Movement 

Oral  Verbalization 

. 

Take-Of  f /Climb /Level 

Off 

Recalling  Procedures 

Non-Verbal  Identification 

Continuous  Movement 

Positioning  and  Serial  Movement 

Oral  Verbalization 

*  Aerobatics 

Recalling  Procedures 

Non-Verbal  Identification 

Continuous  Movement 

* 

Landing 

Recalling  Procedures 

Non-Verbal  Identification 

Continuous  Movement 

Positioning  and  Serial  Movement 

Oral  Verbalization 

Figure  B-6.  Example  of  Using  Training  Stage  and  Training  Tasks 


Figure  B-7,  taken  from  TAEG  Report  No.  1,  illustrates  the  format  for  the  media 
selection  matrices  developed  for  each  primary  task  category.  The  matrix  shows  a  direct 
relationship  between  the  characteristics  of  the  task,  the  learning  guidelines,  the  implica¬ 
tions  for  media  selections,  and  the  media  options  available  from  the  pool.  These 
matrices,  with  modifications  to  ensure  they  are  accurate  and  aviation  training  specific, 
may  be  more  useful  in  identifying  media  candidates  than  the  job  aids  provided  in  Step  2. 

Automated  Instructional  Media  Selection  (AIMs)  Model 


The  AIMs  Model  was  developed  by  Instructional  Science  and  Development,  Inc.  under 
contract  N61339-79-C-0104  to  the  Navy  Training  Equipment  Center.  The  model  was 
designed  and  implemented  as  part  of  an  automated  system  to  facilitate  the  instructional 
systems  development  (ISD)  process,  although  it  is  fully  capable  of  operating  in  a  stand¬ 
alone  mode.  The  AIMS  model  provides  rapid  access  to  a  media  pool  containing  up  to  99 
media  and  99  instructional  attributes.  The  media  pool  is  completely  user-definable. 
Media  pools  are  changeable  and  updatable.  Ail  functions  of  the  system  allow  the  user 
to  tad  access  to  all  capabilities  without  the  need  for  computer  science  or  programming 
expertise. 

Operation 


As  stated  by  Simpson  and  Kribs  (1980),  the  most  unique  feature  of  the  AIMS  model  is 
that  the  user  can  begin  by  using  either  a  predetermined  media  pool  or  by  creating  an 
entirely  new  pool.  Creation  of  a  media  pool  is  a  three-step  process.  The  first  step  is  to 
determine  the  media  to  be  considered.  The  user  should  make  a  list  of  all  the  media,  or 
combinations  of  media,  to  be  included  in  the  pool.  It  is  advisable  to  allow  for  a  wide 
range  of  alternatives.  If  need  be,  the  pool  can  easily  be  narrowed  if  a  shorter  version  is 
desired  for  a  given  project.  Figure  B-8  shows  a  sample  media  listing  that  includes  both 
cognitive  and  hands-on  type  media. 

The  second  step  is  to  determine  the  instructional  characteristics,  or  "attributes,"  that 
can  be  a  difficult  task.  Figure  B-9  presents  a  sample  list  of  attributes.  Note  how  the 
attributes  are  clustered  into  groups  of  similar  characteristics:  (1)  display  characteristics, 
(2)  response  mode,  (3)  evaluation  mode,  (4)  feedback,  and  (5)  special  requirements. 
Breaking  the  attributes  into  small  categories  like  this  simplifies  the  task  of  creating  the 
media  pool  and  helps  make  the  table  itself  a  bit  more  legible. 

The  third  step  is  to  cross-reference  media  and  their  attributes.  For  each  medium 
listed,  it  is  necessary  to  weight  its  ability  to  deliver  each  instructional  characteristic. 
The  weighting  is  made  with  a  single  number  between  0  and  5.  A  "0"  weighting  indicates 
that  the  medium  is  not  capable  of  delivering  an  instructional  attribute.  A  "5"  indicates 
that  the  medium  is  very  capable  of  handling  a  particular  characteristic.  Note  in  Figure  B- 
10  that  the  medium  "audio-tape"  is  ranked  "5"  under  the  attribute  "audio- voice," 
indicating  the  strong  relationship  between  the  two.  "Workbook,"  of  course,  is  ranked  "0" 
under  the  audio-voice  capability,  since  it  is  not  able  to  deliver  this  characteristic. 

Once  the  media  pool  has  been  created,  it  is  entered  into  the  computer  as  a  data  base. 
The  AIMS  system  provides  programs  to  allow  easy  entering  of  these  data.  After  that  task 
is  complete,  the  user  creates  selection  worksheets  for  matching  instructional  character¬ 
istics  to  objectives.  The  system  provides  for  automatic  generation  of  such  a  worksheet. 
The  worksheet,  of  course,  is  directly  tailored  to  the  media  pool  from  which  it  was 
created.  Figure  B-ll  is  a  sample  selection  worksheet  produced  by  the  computer.  The 
instructional  designers/developers  use  the  worksheets  to  indicate  the  instructional  charac¬ 
teristics  required  for  effective  and  efficient  delivery  of  instruction.  Since  this  is  a 
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MEDIA  LISTED  ON  THE  FILE  :  pool 


1.  CAI  •  • 

2.  Random  Access  Slides 

3.  Motion  Picture/  Videotape 

4.  Slides/Audio 

5.  Audio/Tape 

6.  Slides  v  ,  . 

7.  Lecture 

8.  Demonstration 

9.  Group  Discussion 

19.  Workbook 

11.  Teaching  Machine 

17.  OP.  Equip. /OP.  Environment 

13.  OP.  Equip. ,/Lab 

14.  Team  Trainer 

15.  Procedural  Trainer 

15.  Part  Task  Trainer 

17.  3-D  Mock-up 

13.  2-D  Mock-up 

19.  TICCIT  +  Videodisc 

20.  PLATO  +  Videodisc 

31.  GETS 

22.  Besseler  Cue-See 


Figure  B-8.  A  sample  media  listing. 
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ATTRIBUTES  LISTED  ON  THE  FILE  :  pool 


*  I.  DISPLAY  CHARACTERISTICS 

1.  Audio  (voice* 

2.  Audio  (ambient* 

3.  Tactile  cues 

4.  Kinesthetic  cues 

5.  Verbal/text 

8.  Pictorial 

7.  Drawings/diagrams 

8.  Motion 

9.  Color  -  - 

*  II.  RESPONSE  MODE 

10.  Verbal/written 

11.  Point-touch-mark 

12.  Manipulate 

13.  Multiple  choice 

*  III.  EVALUATION 

14.  Instructor 

15.  Automated 

16.  Peer 

17 .  Intrinsic 
♦IV.  FEEDBACK 

18.  Immediate  response 

19.  Immediate  error 

20.  Periodic 

21.  Post  session 

*  SPECIAL  REQUIREMENTS 

22.  Crew/team  interaction 

23.  Environmental  condition 

24.  Physical  motion 

25.  Time  variability 


Figure  B-9.  A  sample  listing  of  instructional  characteristics 
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BgggggBBBBggnmBiBnggBgLn 


1.  CAI 


Random-Access 


3.  Motion  Pic/VT 


Slides/Audio 


Audio  Tape 


Slides 


Lecture 


i— mbhhb  w—ni  inmni  ibbbbihimi 
iMHHHmHHHHi  innnni  innnni  ibbbhi  innm 


9.  Group  Discuss. 


10.  workbook 


11.  Teaching  ftoctiine 


12.  OP. £0UJ?/0P.£« 


13.  OP.  EQUIP/LAB 


14.  Team  Trainer 


Part  Task  Train. 


17.  3-0  Mock  Up 


18.  2-0  Mock  Up 


•  Potential  CAI  * 


T1CCIT  i  VOISC 


.  PLATO  l  VDISC 


II.  GETS 


:Also  Possible: 


12.  Besseler  Cue- 


iinnnnnnnnni  innimi  innnni  gmiinn 
nnaggrannani  innnni  innnni  mmbbmbi  linnni 
fcMMin— i  innnni  infinKi  ■— 


innniwinnnnni  innnni  innnni  ffiEESS-H!!!!! 

—  innnni  innnm  imsmm  nnni 

i  wnn«  innnnrinnni 


nnnnnnni  mnnnunnnn 

2BB5225I  JSSSH  IS9S9 


hbi  mnani  nmnni  sannnrisana 
am  ibuhb!  innsai  ihbbhi  esaa 


Figure  B-10.  A  sample  completed  media  pool  with  each  medium's 
ability  to  deliver  each  characteristic  rated  on  a  0-to-5  scale 
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SELECTION  WCRK3HEET  FOR  USE  WITH  THE  FILE  :  pool 

OBJECTIVE  NUMBER  :  _ . _ 

OBJECTIVE: 


Put  a  check  in  the  boxes  next  bo  required  attributes. 


*  I.  DISPLAY  CHARACTERISTICS 

I _ I  1.  Audio  (voice) 

I _ I  2.  Audio  (ambient) 

I _ I  3.  Tactile  cues 

I _ I  4.  Kinesthetic  cuss  • 

I  I  5.  Vterbal/text 

I _ I  5.  Pictorial 

I _ I  7.  Drawing  s/d  lag  ram3 

I _ !  8.  Motion 

I  I  Color 

*  lT7  RESPONSE  MODE 

I _ I  10.  Verbal/written 

I _ I  11.  Point- touch-mark 

I _ I  12.  Manipulate 

1 _ I  13.  Multiple  choice 


*111.  EVALUATION 

I _ I  14.  Instructor 

i _ I  15.  Automated 

! _ I  Ifi.  Peer 

1  I  \7.  Intrinsic 

*  ITT  FEEDBACK 

I _ I  18.  Immediate  response 

I _ I  19.  Limed  iate  error 

I _ I  20.  Periodic 

I _ I  21.  Past  session 

*  SPECIAL  REQUIREMENTS 

I _ I  22.  Crew/team  interaction 

I _ |  23.  Environmental  condition 

I _ I  24.  Physical  motion 

I  |  25.  Time  variability 


Figure  B-ll.  A  sample  selection  worksheet  from 
a  media  pool  entitled  "POOL." 
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nental  task,  the  worksheets  allow  designers  to  work  "off-line,"  not  consuming 
cessary  computer  time  at  this  stage. 

Once  all  of  the  instructional  characteristics  for  a  given  group  of  objectives  have  been 
rmined,  these  data  need  to  be  entered  into  the  computer.  Basically,  the  person 
tting  data  types  in  the  objective  number  and  the  numbers  of  the  instructional 
butes  checked  off  on  the  worksheet  for  that  objective.  The  computer  then  goes  to 
;  in  several  ways: 

1.  Determines  media  that  have  the  required  characteristics. 

2.  Determines  how  well  each  medium  handles  required  characteristics. 

3.  Rank  orders  media  as  to  their  appropriateness  for  the  given  objective. 

4.  Records  how  many  times  each  medium  is  selected. 

5.  Stores  objectives  with  selected  media  on  data  base  of  objectives. 

Figure  B-12  shows  a  sample  media  selection  made  by  the  AIMS  model.  Note  that  the 
la  are  ranked  according  to  specific  rating.  This  is  a  rating  of  how  well  this  medium 
les  the  characteristics  specific  to  the  objective.  The  general  rating  is  the  average  of 
quality  ratings  assigned  to  all  of  the  instructional  characteristics  for  that  medium. 

Goals 


Simpson  and  Kribs  (1980)  state  that  the  AIMS  model  was  designed  to  meet  five  major 
5.  These  goals,  and  how  they  were  met,  are  described  below: 

1.  Maximize  the  use  of  information  pertinent  to  media  selection.  Some  models 
mpt  simplification  of  the  media  selection  process  on  the  grounds  that  it  is  unwieldy, 
cult  for  those  without  extensive  media  expertise,  time  consuming,  or  too  soft  a 
nology.  The  AIMS  model  approaches  this  process,  first  by  providing  computerized 
ss  to  the  media  pool.  The  media  pool  can  theoretically  be  of  any  size  without 
iasing  the  human  labor  involved  in  the  selection  task.  Second,  a  media  pool  can  be 
ted  by  a  single  person,  or  small  group  of  personnel,  with  extensive  media  expertise. 

maximizes  the  use  of  their  time  and  knowledge.  Matching  training  objectives  to 
uctional  characteristics  is  a  task  that  can  be  performed  by  subject-matter  experts 
or  instructional  development  personnel.  There  is  no  need  for  these  persons  to  have 
ific  media  expertise  at  this  point,  thus  making  full  use  of  the  time  and  knowledge  of 
5  personnel.  Clerical  personnel  can  then  input  data  into  the  computer,  and  it  can 
:t  the  appropriate  media,  rank  order,  record,  and  file  them.  The  bulk  of  the  labor, 
,  actually  falls  on  the  computer.  The  computer  performs  these  tasks  at  a  mere 
tion  of  the  time  required  of  a  human,  and  virtually  makes  no  mistakes. 

2.  Provide  flexibility  in  modifying  the  model  to  reflect  specific  problems  and 
Ring  technology  of  media  or  media  guidelines.  The  recurring  problem  with  current 
ia  selection  models  is  the  broadness  of  media  selections  and  the  rigidity  of  media 
s.  The  possibility  of  creating  a  universal  media  pool  seems  nil.  Media  pools  are  going 
iry  in  the  media  to  be  considered,  relevant  instructional  characteristics,  and  level  of 
ificity  of  media  and  characteristics.  They  are  going  to  have  to  change  with  changing 
nologies,  also.  The  AIMS  model  provides  for  this.  Without  any  special  programming 
rtise,  a  user  may  add  media,  delete  media,  add  attributes,  or  create  entirely  new 
la  pools,  without  compromising  the  selection  process  itself. 

3.  Allow  greater,  more  detailed  selection  of  training  devices,  beyond  the  classroom 
learning  center  type  media.  Since  each  user  is  the  author  of  his  own  media  pool, 
nsive  listings  of  simulators,  realia,  and  other  "hands-on"  type  media  and  their 
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Figure  B-12.  A  sample  media  selection  made  by  the  AIMS  model. 


attributes  are  fully  permissible.  Fidelity  requirements  can  be  included  as  instructional 
characteristics,  thereby  directly  bearing  on  the  media  selected.  The  bias  toward 
"cognitive"-type  media  apparent  in  earlier  models  is  eliminated. 


4.  Provide  consideration  of  stage  of  learning.  To  take  stage  of  learning  into 
consideration,  one  needs  simply  to  incorporate  this  as  an  instructional  characteristic  for 
the  computer  to  select. 

5.  Allow  for  variations  in  the  stage  of  front-end  analysis.  The  conventional 
thinking  behind  media  selection  models  is  that  they  are  used  by  a  combination  of 
instructional  designers  and  people  with  media  expertise  during  instructional  development. 
Usually  included  in  the  "use"  scenario  is  the  asumption  that  a  job/task  analysis  has  been 
performed  with  objectives  defined  and  arranged  in  a  fashion  indicating  instructional 
sequence.  .  .  .  Media  selections  may  be  as  general  or  as  specific  as  available  knowledge 
about  potential  instructional  characteristics  permits,  (pp.  63-64) 

Delphi  Technique 

The  Delphi  technique  is  a  set  of  procedures  for  eliciting  and  refining  the  opinions  of  a 
panel  of  experts.  The  objective  of  the  technique  is  to  obtain  a  consensus  of  expert  opinion 
on  the  topic  under  consideration.  This  requires  determining  the  items  on  which  the 
experts  are  in  agreement.  If  the  responses  of  the  experts  are  randomly  distributed  among 
several  alternatives  for  a  given  topic,  then  no  agreement  exists.  A  statistical  definition  is 
required  to  specify  precisely  what  is  meant  by  the  term  consensus  of  opinion.  Some  users 
of  the  Delphi  technique  have  concluded  that  consensus  exists  if  50  percent  of  the  experts 
agree,  while  other  users  have  employed  measures  of  variability  and  other  sophisticated 
measures.  Two  variables  are  crucial  in  defining  consensus:  the  number  of  experts  on  the 
panel  and  the  number  of  alternatives  available  for  a  given  topic.  If  the  distribution  of  the 
responses  is  likely  to  occur  by  chance,  then  consensus  of  opinion  cannot  be  said  to  exist 
(Herrick,  Wright,  &  Bromberger,  1977). 

This  approach  is  considered  superior  to  the  panel  discussion  for  achieving  consensus 
(Helmer,  1966).  The  procedures  are  designed  to  minimize  (1)  the  effect  of  a  dominant 
(but  not  necessarily  more  expert)  individual,  (2)  the  "bandwagon"  effect,  (3)  the  hesitancy 
to  abandon  a  publicly  expressed  position  once  it  has  been  refuted,  and  (4)  the  pressure  that 
exists  to  renounce  an  unpopular  position  (Keller  &  Koen,  1976).  The  procedures  also  offer 
the  advantage  of  providing  the  expert  with  the  opportunity  to  leisurely  examine  the 
opinions  of  other  experts  and  benefit  from  the  exposure  to  opinions  from  other  experts 
with  different  orientations  and  biases. 

The  Delphi  technique  is  based  on  a  series  of  carefully  designed  questionnaires 
(typically  three  to  five)  interspersed  with  information  and  opinion  feedback  from  the 
previous  iterations.  The  first  questionnaire  consists  of  general  questions  designed  to 
evoke  discussion  of  the  selected  topics  by  the  experts.  In  their  responses,  they  identify 
the  areas  of  relevance  and  importance,  describe  their  knowledge,  provide  examples  and 
experiences  to  support  their  opinions,  suggest  problems  and  offer  solutions.  The  responses 
to  the  first  questionnaire  are  summarized,  statistically  analyzed,  organized,  and 
converted  into  individual  items  for  the  second  round  questions.  Thus,  the  experts 
determine  the  content  of  the  second  and  subsequent  rounds  (Herrick  et  al.,  1977).  The 
iterations  or  rounds  allow  the  panel  to  reconsider  factors  they  might  have  inadvertantly 
neglected  and  to  give  weight  to  those  they  may  have  initially  dismissed  as  inimportant. 

In  summary,  the  Delphi  technique  is  characterized  by  (1)  the  anonymity  of  responses 
through  the  use  of  mailed  questionnaires,  (2)  the  respondents'  expertise,  (3)  requirement  of 
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multiple  rounds  to  obtain  consensus,  and  (4)  controlled  feedback  on  responses  from 
previous  rounds  (Armstrong,  1978). 


In  general,  these  characteristics  may  be  considered  relevant  and  useful  to  the  media 
selection  process.  However,  some  refinements  and  guidelines  are  required  to  make  the 
technique  operational.  Specifically,  the  Delphi  technique  is  too  time-consuming  a  process 
for  making  media  selection  decisions  within  most  ISD  development  efforts.  The 
traditional  use  of  several  rounds  of  mailed  questionaires  with  the  associated  analysis  and 
determination  of  consensus  on  items  may  take  weeks  or  months.  The  statistical  definition 
of  consensus  of  opinion  for  items  may  present  a  problem  in  media  selection.  There  are 
the  issues  of  defining  consensus  to  permit  identification  of  one  or  more  candidate  media 
from  the  available  pool,  and  to  ensure  that  agreement  is  reached  so  that  candidate  media 
can  be  selected  for  each  training  objective. 

For  these  reasons,  the  E-T-E  (estimate-talk-estimate)  method,  which  is  similar  to  the 
Delphi  technique,  is  suggested  for  media  selection.  The  general  characteristics  of  E-T-E, 
as  well  as  the  specific  requirements  for  use  in  selecting  media,  are  provided  below. 

Estimate-Talk-Estimate  (E-T-E) 


E-T-E  is  a  method  to  structure  group  discussions  for  decision-making  purposes.  It 
possesses  the  four  general  characteristics  of  the  Delphi  technique:  (1)  anonymity  of 
responses  is  emphasized,  (2)  the  respondents  are  experts,  (3)  more  than  one  round  is 
required  to  obtain  consensus  for  making  decisions,  and  (4)  controlled  feedback  from 
previous  rounds  is  provided.  The  primary  advantage  of  E-T-E  over  Delphi  designs  is  speed 
of  responses.  The  process  can  be  completed  in  hours  or  days  rather  than  weeks  or  months 
(Armstrong,  1978).  Also,  the  considerations  associated  with  statistically  defining  consen¬ 
sus  are  less  involved,  since  the  emphasis  is  placed  on  making  specific  decisions  rather  than 
achieving  consensus  or  agreement  on  items. 

E-T-E  procedures  call  for  independent  and  anonymous  estimates,  then  a  face-to-face 
group  discussion  (rather  than  mailed  questionnaires),  and,  finally,  another  round  of 
individual  estimates.  Usually,  both  estimate  periods  are  made  anonymously,  and  partici¬ 
pants  are  also  encouraged  not  to  reveal  or  argue  their  own  positions  during  the  talk 
period.  The  initial  estimate  period  approximates  the  first  round  in  the  Delphi.  It  is 
structured  to  evoke  a  set  of  parameters  and  discussion  of  the  selected  topics  by  the  panel 
of  experts.  The  responses  to  the  estimate  period  are  summarized,  organized,  and 
converted  into  discussion  points  for  the  talk  period.  There  are  two  methods  for 
conducting  the  group  discussion:  the  nominal  group  meeting,  in  which  control  is  exerted 
over  the  discussion,  and  the  general  discussion,  in  which  experts  may  argue  for  their  own 
positions.  Likewise,  there  are  two  methods  for  conducting  the  last  estimate  period:  the 
pooled  estimate  of  anonymous  experts  to  reach  a  decision,  or  a  group  decision. 

Given  these  general  characteristics  of  the  E-T-E  method,  the  following  subsection 
provides  considerations  and  guidelines  for  its  use  as  a  media  selection  technique  in 
military  technical  training  settings. 

Media  Selection  Using  E-T-E 

One  of  the  most  important  considerations  in  using  the  E-T-E  method  as  a  media 
selection  tool  is  to  establish  selection  criteria  for  the  panel  of  experts.  Since  media 
selection  decisions  have  operational  and  cost  implications  for  a  number  of  activities  in 
Navy  training  commands,  it  is  suggested  that  the  panel  of  experts  be  representative  of 
these  activities.  Guidelines  for  selecting  panel  should  specify  at  least  one  expert  from 
each  of  the  following  personnel  types: 
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1.  Program  management  personnel  concerned  with  planning,  programming,  and 
budgeting  for  the  training  program  under  consideration. 

2.  School  adminstration  personnel  concerned  with  operation  and  maintenance  of  the 
training  program. 

3.  Subject-matter  experts  in  the  content  areas  to  be  trained. 

4.  Experienced  training  system  designers. 

5.  A  media  specialist. 

The  five  variables  listed  below  are  considered  essential  to  any  media  selection 
technique: 

1.  Practical  factors  and  training  program  constraints. 

2.  Trainee  characteristics. 

3.  Training  task  requirements. 

4.  Instructional  characteristics  (stimulus,  response,  feedback  requirements). 

5.  Media  attributes. 

These  variables  must  be  addressed  in  the  three  rounds  prescribed  in  the  E-T-E 
method.  The  first  estimate  period  includes  all  five  of  the  variables  since  it  establishes 
the  parameters  for  selecting  media  for  the  specific  training  program  under  consideration. 
The  talk  period  provides  an  opportunity  to  analyze  the  match  of  instructional  character¬ 
istics  required  to  support  the  training  tasks  with  the  media  attributes  of  the  candidates 
available  in  the  media  pool.  The  last  estimate  period  focuses  on  trade-offs  and  practical 
factors  in  order  to  make  the  media  selection  decisions  required  for  the  training  program. 
The  considerations  for  each  of  these  rounds  are  described  in  more  detail  below. 

The  first  round  is  critical  in  using  E-T-E  for  media  selection.  Any  training  program 
constraints  or  significant  practical  factors  that  may  impact  the  media  decisions  should  be 
identified  early  in  the  process.  These  may  include  such  things  as  command  policy,  phase 
in  the  weapon  system  and  training  program  life  cycle  (i.e.,  conceptual  or  operational),  and 
training  resources  availability  (i.e.,  facilities,  funding,  instructional  staff,  or  support 
personnel).  These  will  establish  the  parameters  for  defining  the  remaining  four  variables. 
The  trainee  characteristics  should  be  reviewed  to  identify  entry  level  behaviors,  student 
throughout  and  attrition  rates,  and  any  other  pertinent  factors.  The  candidates  included 
in  the  media  pool  should  be  examined  for  relevancy  and  adequacy  to  meet  the  training 
requirements  of  the  program  under  consideration.  The  media  attributes  of  the  candidates 
to  be  included  in  the  media  pool  should  be  thoroughly  defined.  The  format  for  the  training 
task  requirements  (i.e.,  training  objectives,  or  stages  of  training  and  tasks)  should  also  be 
determined.  In  addition,  the  procedures  for  matching  the  instructional  characteristics  of 
these  training  requirements  to  the  attributes  first  estimate  period. 

The  procedures  for  conducting  the  first  round  might  be  as  follows: 

1.  Each  expert  on  the  panel  receives  a  list  of  preliminary  definitions  for  each  of  the 
five  media  selection  variables  for  review  and  critique. 

2.  The  panel  makes  detailed  definitions  for  each  variable  and  returns  them  to  the 
coordinator. 
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3.  The  coordinator  analyzes  and  summarizes  the  responses  to  determine  agreement 
on  the  definitions  of  the  variables. 

Sufficient  agreement  must  be  achieved  during  the  first  estimate  period  in  order  to 
perform  the  match  of  instructional  characteristics  against  media  attributes.  In  fact,  it 
may  be  advantageous  in  some  cases  to  include  a  second  pass  during  the  initial  round  to 
accomplish  this  task  prior  to  the  group  meeting,  especially  if  there  are  time  constraints 
on  the  panel  of  experts. 

The  results  of  the  first  round  are  then  used  to  structure  the  face-to-face  group 
meeting.  The  meeting  may  be  conducted  in  either  one  of  the  two  ways  described  earlier. 
The  primary  purpose  of  the  face-to-face  meeting  is  to  identify  one  or  more  candidate 
media  for  each  training  objective  or  training  task  within  a  training  stage.  Sufficient 
agreement  must  be  achieved  during  the  group  discussion  to  accomplish  this  goal.  If  the 
match  of  instructional  characteristics  to  media  attributes  was  performed  as  a  second  pass 
in  the  first  round,  this  should  be  a  review  process.  If  it  is  to  be  performed  as  a  group, 
ample  time  must  be  allowed  for  discussion. 

The  results  of  the  second  round  are  then  used  to  make  trade-offs,  reconsider  initial 
constraints,  and  identify  any  other  constraints  or  practical  factors  that  might  impact  the 
final  selection.  Once  again,  the  final  round  may  be  conducted  in  a  group  mode  or  as  a 
pooled  estimate  of  individual  expert  responses.  This  depends  upon  the  nature  of  the 
decisions  to  be  made,  and  the  time  constraints  on  the  panel  of  experts  and  coordinator. 
The  desired  end  product  of  the  three  E-T-E  rounds  is  the  selection  of  an  optimal  and 
usable  media  mix  to  meet  the  requirements  of  the  training  program  under  consideration. 
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